feat(release-audit-report): add structured audit-record schema and schema-violation eval#722
Merged
Conversation
…hema-violation eval Add `skills/release-audit-report/audit-record-schema.md` — a canonical definition of required vs optional audit-record fields, the schema-violation reporting contract, and the privacy boundary for each field. Update `release-audit-report/SKILL.md` to reference the schema in Step 2, add `schema_violations` to the Step 2 JSON output contract, and extend the Step 4 hand-back artefact to surface violations to the RM. Extend the eval suite (step-2-assemble-record) with a `has_schema_violations_consistent` judge assertion across all cases, update the output-spec to document the new field, add `schema_violations` to the three existing expected.json files, and add `case-4-all-required-missing` — a fixture that proves the skill reports all nine required-field violations when none of the lifecycle evidence was recorded on the planning issue. Generated-by: Claude (Opus 4.7)
potiuk
approved these changes
Jul 4, 2026
potiuk
left a comment
potiuk
left a comment
Member
There was a problem hiding this comment.
LGTM — behaviour change (Step 2 now emits a schema_violations list) ships with its matching eval: a canonical audit-record-schema.md, a new case-4-all-required-missing fixture, and a has_schema_violations_consistent judge assertion. The fixtures are internally consistent with the schema and the judge rubric. CI green.
Non-blocking nit: the README/case-4 wording says "nine required fields" while audit-record-schema.md lists ten required rows (the tenth, version, is always supplied from the trigger arg so never surfaces as MISSING) — worth tidying the wording on a later pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
skills/release-audit-report/audit-record-schema.md— a canonical definition of required vs optional audit-record fields, the schema-violation reporting contract, and the privacy boundary for each field. Updaterelease-audit-report/SKILL.mdto reference the schema in Step 2, addschema_violationsto the Step 2 JSON output contract, and extend the Step 4 hand-back artefact to surface violations to the RM.Extend the eval suite (step-2-assemble-record) with a
has_schema_violations_consistentjudge assertion across all cases, update the output-spec to document the new field, addschema_violationsto the three existing expected.json files, and addcase-4-all-required-missing— a fixture that proves the skill reports all nine required-field violations when none of the lifecycle evidence was recorded on the planning issue.Generated-by: Claude (Opus 4.7)
Type of change
.claude/skills/<name>/) — eval fixtures updated belowtools/<system>/*.md)tools/*/withpyproject.toml)docs/,README.md,CONTRIBUTING.md)projects/_template/)prek, workflows, validators)Test plan
prek run --all-filespassesuv run pytest/ruff check/mypypasses(
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)(a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)